A SAN is used to interconnect nodes within a distributed computer system, such as a cluster. The SAN is a type of network that provides high bandwidth, low latency communication with a very low error rate. SANs often utilize fault-tolerant technology to assure high availability. The performance of a SAN resembles a memory subsystem more than a traditional local area network (LAN).
The preferred embodiments will be described as implemented in the ServerNet™ (ServerNet) architecture, manufactured by the assignee of the present invention, which is a layered transport protocol for a System Area Network (SAN). The ServerNet II™ protocol layers for an end node and for a routing node are illustrated in FIG. 1. A single NIC and session layer may support one or two ports, each with its associated transaction, packet, link-level, MAC (media access) and physical layer. Similarly, routing nodes with a common routing layer may support multiple ports, each with its associated link-level, MAC and physical layer. The link layer protocol provides link management functions, encoding and decoding of data and command, and buffering of received packet data. The ServerNet II™ link layer protocol is a set of simple protocols, running concurrently to manage the flow of status and packet data between ports on independent nodes. Each port contains a transmitter (TxPort) and a receiver (RxPort) which cooperate to manage the link.
Support for two ports enables a ServerNet™ SAN to be configured in both non-redundant and redundant (fault tolerant, or FT) SAN configurations as illustrated in FIG. 2 and FIG. 3. On a fault tolerant network, a port of each end node may be connected to each network to provide continued message communication in the event of failure of one of the SANs. In the fault tolerant SAN, nodes may be also ported into a single fabric or single ported end nodes may be grouped into pairs to provide duplex FT controllers. The fabric is the collection of routers, switches, connectors, and cables that connects the nodes in a network.
The SAN includes end nodes and routing nodes connected by physical links. End nodes generate and consume data packets. Routing nodes never generate or consume data packets but simply pass the packets along from the source end node to the destination end node.
Each node includes one more full duplex ports each connected to a physical link. A link layer protocol (LLP) manages the flow of status and packet data between ports on independent nodes.
The ServerNet™ SAN has the ability to perform system management from a single point anywhere in the SAN. SAN management performs many functions including collection of error information to isolate faults to the link or module where the faults occurred.
An “In Band Control” or IBC mechanism supports a low overhead way of performing SAN management functions. The term “in band” indicates that the network management control data travels over the existing SAN links—with no separate cable or LAN connection. In contrast to data packets, both routing nodes and end nodes generate and consume IBC packets. IBC packets are not routed like data packets, each IBC packet contains embedded source routing information. Each router or end node that receives the IBC packet forwards it to the next destination in source route list.
The ServerNet™ SAN includes a maintenance system having responsibility for system initialization, fault reporting, diagnostics, and environmental control. A pair of service processors (SPs) manage the maintenance system. The SPs function as ServerNet™ I/O controllers and communicate with each other only via the ServerNet™ SAN.
The maintenance system uses dual system-maintenance buses which form redundant trees, independent of normal system functional paths and provide a path of two industry standard interconnects. The maintenance system controls, initializes, tests, and monitors all ASIC operations and provides a means for ASIC initialization, SAN topology determination, and error reporting.
One function of the link layer protocol is to maintain a “this link bad” (TLB) protocol. TLB (this link bad) commands are used to communicate exception status between the nodes comprising a link. Status flags and other state information are used to communicate exception status to software either directly (far end nodes) or via the maintenance system (for routing nodes). A port transmits a TLB command when the link is alive and one of the exceptions listed in the table of FIG. 4 occurs. Exceptions listed in FIG. 4 include: receive FIFO overflow; packet CRC exception; packet length exception (run-on packet); packet framing exception; link alive exception; and command exception.
The receive FIFO overflow type of exception indicates that the receive FIFO was full when new packet data was received and that the new data has been discarded. The packet CRC type of exception indicates that a packet framed by a TPG command failed the CRC check. The packet length type of exception indicates that the receive link layer protocol logic has processed a packet with a data link greater than 1,796 data bytes. The packet framing type of exception indicates that a packet terminated by an IDLE command was received. The link alive type of exception indicates that the link alive status has changed state, e.g., from the status of link alive to link dead. The command type of exception indicates an illegal or unsupported command is received or that a TLB or OLB command is received. Note that one of the conditions that results in transmission of a TLB command is the receipt of a TLB command. Thus all link errors result in the transmission of TLB commands in both directions on a link.
The ServerNet™ SAN has been enhanced to improve performance. The original ServerNet™ SAN configuration is designated SNet I and the improved configuration is designated SNet II. Among the improvements implemented in the SNet II SAN is a higher transfer rate and different symbol encoding. To attach SNet I end nodes and routing nodes to serial cables, a special two-port router ASIC matches SNet I devices to SNet II devices. This two-port router will be referred to as a “link-extender”, as used herein, is only a convenient name and does not connote any limitations on the functioning of the device.
A typical connection utilizing link-extenders is depicted in FIG. 5. Node A is connected to first link extender x by link 1. The first link extender x is coupled: by link 2 to a second link extender y. The second link extender y is coupled by link 3 to the Node B. The link extenders normally operate without intervention of system error handling software. Accordingly, the system error handling software sees the connection as a single link and is not able to isolate faults to a particular physical link, e.g., link 1, 2, or 3 in FIG. 5.